Salih Cepni , 1 Yilmaz Kara , 1 and Emine Cil 3 


Journal of Testing and Evaluation, Vol. 40, No. 3 
Paper ID JTE104274 
Available online at: www.astm.org 


Middle School Science and Items of High School 
Entrance Examination: Examining the 
Gap in Turkey 


ABSTRACT: This study presents findings from an analysis of the Turkish Science and Technology Curriculum Guidelines and their alignment 
to the university entrance examination. The analysis of the Science and Technology Curriculum focused on various related aspects: content areas 
and learning outcomes in terms of scientific process skills, science technology society and environment, attitudes and values, and cognitive objec- 
tives. The level determination examination (LDE) questions, which would allow for enrollment at elite high schools and would inevitably affect 
middle school students’ university choices, were analyzed in an alignment study regarding the contents of the science and technology program. In 
order to investigate the relationship between high school science teaching and high school entrance examination, all relevant documents were ana- 
lyzed by ten science teachers, who have been teaching science and technology courses at middle schools and are concurrently doctoral students at 
the science education department. The results indicated that the LDE questions and standards were not fully aligned because the first two alignment 
criteria have a high consistency while a range of knowledge and balance of representation criteria have a low consistency. 
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Introduction 

In many countries, public high schools have become increasingly 
attractive to students because their main goal is to place students 
in a qualified university. According to Sahlberg [1], young stu- 
dents’ attitudes towards technical high schools in Organisation for 
Economic Co-Operation and Development (OECD) countries are 
generally not positive, preferring instead to enroll in public high 
schools where they would get more opportunities to continue at 
university level. Because of the limited quota for entering quali- 
fied high schools, there is tough competition among students who 
want to enter these schools, which has continued for many years. 
It is known that many countries use different testing or alternative 
measurement and assessment systems at these types of entrance 
examination. The general infonnation and description of middle 
and high school entrance examination systems or exit examina- 
tions for Cambodia, Indonesia, Philippines, Singapore, Thailand, 
Vietnam, China, Holland, and England have been reported in 
various studies [2-5]. In most of these countries, mathematics, 
science, and language lessons make up the main parts of these 
tests, which mostly consist of multiple choice questions aiming to 
measure students’ cognitive levels or cognitive capacities [6]. 
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Recently, many researchers or policy makers in education have 
been interested in the examination systems of other countries or 
more specifically in what types of questions are being asked in 
nationwide middle-secondary schools or university entrance 
examinations. Some of these studies have investigated students’ 
achievement scores in such exams [7-10], while others have 
investigated whether there is any significant difference between 
male and female students and what influence(s) students’ achieve- 
ment in these examinations [1 1,12], Some researchers have inves- 
tigated possible factors that could influence students’ success in 
entrance exams [13,14], Some studies have looked for the relation 
between students’ entrance examination scores and their academic 
achievement. These studies have tried to determine whether it 
would be possible to measure students’ behaviors or real abilities, 
which the new curricula require, with central examinations 
[15,16]. Some studies have mostly focused on the reliability of the 
questions in the existing curriculum and the cognitive levels of 
questions asked in these examinations [5,17-22]. 

A big debate has also started concerning the influence of cen- 
tral examinations on students’ psychology and their negative 
effects on teaching, the students’ learning process at schools, and 
the weakness of examinations in determining students’ real abil- 
ities. For example, according to Sternberg [23], classic tests are 
able to measure students’ analytic thinking or analytic capacities. 
But these types of tests or examination systems only focus on one 
type of intelligence. They mostly ignore students’ creativity and 
practical abilities. In contrast, WoBmann [24] argued that multiple 
tests, based on central examinations and graduation examinations, 
have positive effects on improving students’ performance. 
Because students are ranked according to the score they get from 
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such examinations, they will try to improve their capacities and 
abilities. In addition, these exams also have positive effects on the 
schools’ quality and teachers’ performance. However, according 
to some researchers, even if students are very good at solving mul- 
tiple test questions, the same students may encounter problems in 
understanding concepts or solving scientific or technological 
problems related to real life situations [25]. Cepni and Cil [9] 
explained this situation in terms of procedural understanding 
being more dominant than conceptual understanding. Many teach- 
ers now know that even without understanding the real meaning 
of concepts, students may still be able to solve the test questions 
by memorizing certain rules which are valid in solving certain 
types of questions. They really do not use critical thinking skills 
and they may not do things in order to solve these questions 
either. Teachers mostly believe that if they emphasize concept 
learning or spend time on discovery learning, their students would 
get low marks on entrance examinations [26-29]. Therefore, it 
may be concluded that the nature of a nationwide examination is 
not consistent with the latest learning theories or active learning 
strategies. Teachers’ achievements are directly related to how 
many questions their students are able to correctly solve in central 
examinations. Apart from the teacher-centered or student-centered 
approach, this new situation adds a new term to the existing litera- 
ture, “exam-centered approach,” in which the main focus is the 
ability to solve as many test questions as possible [30,31]. In 
many cases, experts in an exam-centered approach would recom- 
mend that students solve 500-1000 test questions per day so as to 
get accustomed to all styles of questions which could be asked in 
central examinations. One could thus observe many students at 
school or after school, “eating toast and at the same time solving 
test questions,” that is, trying to learn the concepts via testing. All 
these habits do not fit in with the nature of a newly developed cur- 
riculum [12,25,32-35], 

As Kasanen and Raty claimed [36], national exams put teach- 
ers in a difficult situation. Because students and teachers need 
to put in effort and time to prepare for the level determination 


examination (LDE)-type exams, they consider projects and per- 
formance works as a matter of formality. In the end, almost all 
teachers came to the same conclusion: “This new program is not 
suitable for our education, especially the assessment system, so 
we need a new one.” In Turkey, many programs collapsed for the 
same or similar reasons [30]. The problem is that “what we should 
do in order to prevent teachers or students from spreading nega- 
tive propaganda about the new curriculum,” and “how we can 
convince them that the contents and assessment approaches in the 
new curriculum are mostly consistent with the nature of Trends in 
International Mathematics and Science Study (TIMSS), Pro- 
gramme for International Student Assessment (PISA), or the 
LDE” is not known. Actually, there have been studies that exam- 
ine the relationship between the curriculum standards/objectives 
and assessments frameworks [21,37]. Standards-based curricu- 
lums, like the Turkish one, impose the alignment between content 
standards and nationwide exam questions [16]. 

The purpose of this study is twofold: ( 1 ) to analyze the content 
areas of the specified objectives of the new science and technol- 
ogy curriculum for grades 6-8 as scientific process skills (SPS), 
science technology society and environment (STSE), attitudes and 
values (AV), and cognitive; and (2) to investigate alignments of 
the LDE questions with the contents of the science and technology 
programs for grades 6-8. 

Contextual Background 

The Turkish education system is highly bureaucratized, central- 
ized, and exam-oriented. All decisions related to educational 
matters follow a top-down procedure. All schools throughout the 
country must use the same curriculum developed and imple- 
mented by the National Ministry of Education. In many cases, 
teacher-centered approaches used in the teaching-learning process 
are still dominant [38]. The present conditions of the Turkish edu- 
cation system are summarized in Table 1 in terms of results from 
external and internal indicators. 


TABLE 1 — External and internal indicators. 


International (external) indicators National (internal) indicators 


PISA Performance on Science Reading Test: Turkey had 434 points in the 
PISA Science Reading Test in 2003. In the next test in 2006, the average score 
among Turkish students was 424 points. At the last one, the PISA authorities 
announced that the average science reading test score was 454 points. 

TIMMS Performance: According to 2007 results, students at eighth grade 
level are ranked 3 1 st among 49 countries. 

The Relevance of Science Education (ROSE): There have been a few 
research concerning students and teachers’ attitudes towards science and tech- 
nology education and their epistemological knowledge and expectation for the 
future. Similarly, for example, the main problems of science (especially 
physics) teaching in Germany were summarized as: students’ lack of interest 
and motivation in the subject, poor understanding of scientific concepts, ideas, 
methods, and results, and their lack of comprehension of the social, political, 
and epistemological role of science (Riess, 2000). 


The Average Scores of the Secondary Education Entrance Exam: The 

average of the Science Lessons Questions is 4.79 questions from 25 science 
questions in 2005. At the last entrance examination which is now called the 
LDE (covering 6 th -8 th grade level curriculum) and took place in 2009, science 
achievement results were 29.8 % at 6th level and 33 % at 7th level. 

Students’ Attitudes Towards Science and Mathematics: Gradually decreas- 
ing starting from elementary schools. 

Students’ Socio-economic Conditions: Changeable, but a few students 
(approximately 5 % of the students) are able to continue to qualified private 
schools. The learning environment in many cases lacks equipped and science 
laboratory, has overcrowded classrooms, and unqualified teachers. 

Teachers’ Attitudes Towards New Curriculum: Recent research shows that 
the majority of teachers show negative attitudes towards the new science 
curriculum. 

The Average Scores of the University Entrance Exam: The average of cor- 
rectly answered science questions were 3,9 out of 45 questions. 
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The PISA survey (measures students’ knowledge in science; 
students’ ability to use scientific knowledge and relate this knowl- 
edge to everyday problems; and students’ performance in reading 
and mathematics; also collects data on the student, family, and 
institutional factors) could help explain the differences seen 
among countries. The survey concentrates for the most part on the 
science content area including the following subjects: health, natu- 
ral supply, environment, hazardous matters, science, and technol- 
ogy. Students were assessed about their proficiency in scientific 
knowledge and attitudes before their graduation from elementary 
schools that have covered these subjects [40], 

Turkey has had a lot of experience with curriculum develop- 
ment up until 2004. These attempts were described as a failure by 
many researchers [30], Starting from elementary school, curricu- 
lum reforms were implemented in all subjects at all levels, includ- 
ing high schools. With the new curriculum, many changes have 
taken place in tenns of learning areas and the secondary level 
school entrance examination system. 

Learning Areas and Units of the New Science and 
Technology Curriculum 

In Turkey, all curriculua are designed by a committee of experts 
at the National Ministry of Education and approved by the Board 
of Education and Discipline. All schools in the nation follow the 
same programs. In the case of science and technology education 
program, the curricula consist of vision, aims, reasoning, instruc- 
tional methods and techniques, measurement and evaluation 
approaches, samples for measurement and evaluation, and materi- 
als for activities besides theoretical information. The program 
document consists of all the units and topic content with their out- 
comes from grades 6 to 8. The new primary science and technol- 
ogy curriculum has four content areas, as follows: 

1 . Physical Events 

2. Life and Living Beings 

3. Matter and Change 

4. The Earth and the Universe 

These content areas are engaged with through the instructional 
objectives of science process skills (SPS), science technology 
society environment (STSE), and attitudes and values (AV). 
These instructional objectives were integrated with the cognitive 
objectives (Fig. 1). 


All the outcomes and learning areas are coded as SPS, STSE, 
AV, or cognitive. For example, an objective of sixth grade out- 
comes is stated as follows in the curriculum: “ Students can give 
examples of chemical change in which a matter transforms into a 
different matter (SPS-6, 8).” However, some outcomes, which are 
addressed with verbs like “define,” “explain,” or “distinguish,” 
are regarded as cognitive outcomes. An example from seventh 
grade is: “ Students can explain that the distance among the stars 
is measured in ‘light year’ units. ” All the SPS, STSE, AV, and 
cognitive outcomes were counted one by one by the author from 
grades 4 to 8. Some outcomes can target more than one learning 
area. For example, an outcome covering both STSE and AV learn- 
ing is: “ Students can emphasize that physically changed matter 
does not change its identity (SPS-6, 8, 9; AV-2)." In such a situa- 
tion, the outcome is counted as both STSE and AV outcome and 
added to the total sum of outcomes. 

System for Secondary School Entrance 

With the new program, the secondary school entrance examination 
system was also changed. Before the change, only one exam used 
to be given, at the end of the eighth grade year. With the new sys- 
tem, students (sixth to eighth grades) start to take this exam at the 
end of each academic year. Even the name of this exam was 
changed. It is now called the Level Determination Examination 
(LDE), which aims to follow up students’ performance each year 
for the executed curriculum. In these exams, students are responsi- 
ble for the following lessons: Turkish, Mathematics, Science, 
Social Science, and English. These examination results make up 
70 % of the final grade, while students’ performance and behaviors 
in the classroom account for 30 % of the students’ total scores. In 
this process, total score of the students is calculated by adding 
25 % score of sixth grade level and 35 % score of seventh grade 
level to 40 % score of eighth grade level [41]. The LDE was first 
applied in the 2007-2008 academic year. Sixth and seventh grade 
students participated in this exam for the first time. Students in 
eighth grade took different exams because they followed the old 
curriculum during the 2007-2008 academic year. The wide usage 
of multiple choice questions in national tests in order to enroll at a 
variety of schools brought up new discussions about the validity of 
such tests. Riffert mentioned two types of validity: curriculum and 
criterion validity [42], Curriculum validity is the similarity between 
the expected outcomes in tests among students and the curriculum 



FIG. 1 — The structure of the new Turkish science and technology curriculum. 
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executed in schools. In Turkey, because all schools implement the 
same curriculum, the validity of the curriculum has to be deter- 
mined from the examinations. In international examinations such 
as TIMMS and PISA, the criterion of validity has to be acquired. 
For instance, PISA evaluates the level of scientific literacy of stu- 
dents. For this study, only science and technology questions from 
the LDE booklets for sixth to eighth grades were analyzed by 
reviewers according to objectives under the standards covering 
cognitive, SPS, STSE, and AV objectives. 


Methodology 

For the first research question, a task analysis of the elementary 
science and technology curriculum was conducted. Task analysis 
is the resultant of necessary skills and types of thinking in solving 
a problem [43,44], In this context, task analysis is vital to explain- 
ing the knowledge and skills included in the curricula or system- 
atic structure of an examination. The document analysis method 
was used in order to analyze the objectives in science and technol- 
ogy programs (6-8) [45]. The document analysis method is 
described according to systematic criteria [20], After determining 
the number of outcomes in each unit, the sum total of the units’ 
outcomes is calculated. Then, the percentages of learning areas 
are sorted out and the results are shown in tables. The general 
analysis for each learning level was accomplished after the unit 
analyses. In order to accomplish the objectives that were added to 
the corresponding learning level, the total number of outcomes, 
which will be dealt with in a one-year process, is determined in 
the same way. Then, the four learning areas of each unit are added 
separately. The percentages of SPS, STSE, AV, and cognitive 
objectives were calculated from the sum of outcomes. 

For the second research question, all the data were analyzed by 
ten science teachers who have been teaching science and technol- 
ogy courses at middle schools in the Black Sea Region and who are 
at the same time pursuing their Ph.D. program at the science educa- 
tion department of the Fatih Faculty of Education. These teachers’ 
teaching experiences amounted to between 3 and 10 years in the 
field. The reviewers were trained by the researcher of this study 
during a Ph.D. course called “Using Learning Theories in Science 
Curriculum,” covering the content analysis of the new science and 
technology curriculum, learning theories, assessment framework, 
and the objectives of science process skills (SPS), science technol- 
ogy society environment (STSE), and attitudes and values (AV). 

It is possible to use different models in order to investigate the 
conformity between curriculum on the one hand and exam ques- 
tions on the other [46,47], The first model is a simple one that is 
used to make comparisons between exam questions and curricu- 
lum topics [5], There is also another more comprehensive model, 
which allows for more detailed analyses. This comprehensive 
model consists of four criteria: (1) categorical concurrence 
(or content consistency), (2) depth-of-knowledge consistency (or 
cognitive demands), (3) range-of-knowledge correspondence (or 
content coverage), and (4) balance of representation (or distribu- 
tion of test items) [37]. In the present study, Webb’s model was 
used with some adaptations [37]. The explanation of these criteria 


and the complete procedure employed in this study in order to ful- 
fill the requirements for each criterion are summarized as follows: 

(1) Categorical Concurrence Criterion 

This criterion gives us general infonnation on whether the assess- 
ments included items for measuring content from each standard. 
Normally, it is to be expected that the questions should cover all 
standards. First, tables for each unit were drawn up covering the 
objectives under each standard, and questions asked in the LDE 
were also placed in the tables. In order to determine whether the 
LDE included items for measuring content from each standard, 
the reviewers were asked to analyze LDE questions for the sev- 
enth grade. In this process, each reviewer coded the selected LDE 
question by linking directly to the objectives under the standard 
by marking when there was a match between the question and the 
objectives of the new seventh grade science and technology pro- 
gram. The acceptable agreement coefficient was identified to be at 
least 0.63 using a procedure developed by Subkoviak [48]. 

(2) Depth-of-Knowledge (DOK) Consistency 
Criterion 

As Webb had argued [37], the DOK consistency between stand- 
ards and assessment indicates alignment between what is elicited 
from students on the assessments as demanding cognitively and 
what students are expected to know and do as stated in the 
standards. Learning areas, unit, standards, objectives under the 
standards, and items of the LDE questions were considered to 
investigate the DOK consistency between the objectives and 
LDE. The LDE questions were categorized as SPS, STSE, AV, 
and cognitive according to the objectives of the two units. The re- 
vised Bloom’s taxonomy was adapted into these criteria [49]. The 
reviewers compared all the objectives under the standards with the 
LDE questions according to the cognitive process dimensions of 
the revised taxonomy. In this process, LDE questions covering 
units 1 and 2 for the seventh grade were analyzed. The structure 
of the revised taxonomy is as follows: 

1. Remember — retrieving relevant knowledge from long-term 
memory 

2. Understand — Determining the meaning of instructional 
messages, including oral, written, and graphic 
communication 

3. Apply — a procedure in a given situation 

4. Analyze — Breaking material into its constituent parts and 
detecting how the parts are related to one another as well as 
their overall structure or purpose 

5. Evaluate — Making judgments based on criteria and 
standards 

6. Create — Putting elements together to form a novel, coherent 
whole or make an original product 

The science and technology teachers were asked to consider 
the DOK levels of the LDE items and categorize concurrent objec- 
tives as compatible, partially compatible, and weakly compatible. 
The minimal acceptable DOK consistency level should be 50 % 
among the items and the coded objectives [37], The DOK level 
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consistency was used for the cognitive objectives and the LDE 
questions covering these objectives. The other LDE questions 
relating to the objectives and belonging to AV, SPS, and STSE 
were also examined in-depth. 

(3) Range of Knowledge Correspondence Criterion 

This criterion mainly concerns what constituted the appropriate 
breadth of coverage for a standard, and it is used to compare what 
the standard required of students to be able to do with the span of 
knowledge that students need to correctly answer the assessment 
items. “ Full compliance ” represents 50 % or above; “ partial com- 
pliance ” represents 40 % to 50 %; and “weak” represents less than 
40 % [37], The two chapters that cover almost 50 % of the LDE 
questions were examined in-depth by the reviewers. The reviewers 
first examined the LDE questions and then compared them with 
the objectives under the standards to judge whether the depth and 
range of knowledge given in the objectives were adequate with 
respect to the span of knowledge that students need to answer the 
questions correctly. Apart from Webb’s requirements for this crite- 
rion, interviews with each reviewer were also done for the purpose 
of augmenting the quantitative data with qualitative findings. 

(4) Balance of Representation Criterion 

Although the first three criteria are able to judge the alignments of 
curriculum standards and assessments from many angles, they do 
not mention how the questions are distributed among the objec- 
tives. The balance of representation criterion is used to indicate 
the degree to which one objective is given more emphasis in the 
assessment than another. An index is used to evaluate the distribu- 
tion of assessment items for each standard [37]. In order to fulfill 
this criterion, the reviewers were asked to decide which LDE 
questions belong to which objective(s) under the standard by com- 
paring the objectives under the standards for the two units of sev- 
enth grade science and technology curriculum. In this index, 
“unacceptable” indicates an index value below 0.60, “weak” indi- 
cates an index value of 0.60 to 0.69, and “acceptable” indicates an 
index value of more than 0.70. 

Findings 

In the first part of the study, the objectives under the standards 
belonging to the new science and technology curriculum programs 
(grades 6-8) were categorized as cognitive, SPS, STSE, and AV 
as follows. In the second part of the study, data obtained from the 
alignment studies were provided in detail. 

Findings Related to the Science and Technology 
Curriculum Programs (6-8) 

The new science and technology curriculum (grades 6-8) takes a 
spiral approach to each learning area. At the beginning, we ana- 
lyzed the new curriculum as objectives (cognitive, SPS, STSE, 
and AV), unit-based for each level (grades 6-8) from the curricu- 
lum guidelines, and data were displayed in Table 2. 

Findings from Table 2 show that the SPS objectives chiefly 
dominate the entire curriculum while the framework of the science 


TABLE 2 — Percentile distribution of the outcomes of science and 
technology curriculum. 


Grade 

Chapter 

Cognitive 

SPS 

STSE 

AV 

6 

1 . Reproduction birth and growth 
in living things 

53 

37 

5 

5 


2. Force and Motion 

38 

62 

- 

- 


3. Particular Structure of the matter 

23 

59 

4 

14 


4. Electricity in our life 

44 

33 

17 

6 


5. Systems in our body 

60 

4 

26 

10 


6. Matter and heat 

26 

44 

4 

26 


7. Light and Sound 

8 

66 

13 

13 


8. What is earth’s outer structure 
composed of? 

26 

36 

29 

9 

7 

1 . Systems in our body 

50 

14 

29 

7 


2. Force and Motion 

32 

46 

14 

8 


3. Electricity in our life 

62 

32 

6 

- 


4. Structure and the properties of matter 

56 

25 

13 

6 


5. Light 

20 

50 

13 

17 


6. Human and environment 

16 

32 

36 

16 


7. Solar System and more: Space Puzzle 

22 

38 

32 

8 

8 

1 . Cell Division and Heredity 

48 

27 

22 

1 


2. Force and Motion 

42 

46 

4 

8 


3. Structure and the Properties of Matter 

35 

35 

15 

15 


4. Sound 

26 

52 

6 

16 


5. States of Matter and Heat 

38 

38 

12 

12 


6. Living Things and Their 
Relation to Energy 

52 

24 

20 

4 


7. Electricity in our life 

50 

33 

13 

4 


8. Natural Process 

10 

40 

40 

10 


teaching program mainly constructed on cognitive objectives. 
According to the percentage distribution of the objectives in the 
curriculum guidelines, SPS objectives are emphasized more at the 
sixth grade level than at the seventh and eighth grade levels. 

Analysis of LDE Science and Technology Questions 

As to the findings regarding the analysis of the seventh grade cur- 
riculum, the LDE science and technology questions asked in 2009 
were given in this section. There were 1 8 questions in the seventh 
grade LDE science and technology section (see the Appendix). 
The dispersion of these questions according to chapters and learn- 
ing areas is given in Table 3. 

When we examined seventh grade LDE questions asked in the 
2008-2009 academic year according to unit categories in the sci- 
ence and technology program, half of the questions were asked 
from only two units, “Systems in Our Body”, and “The Structure 
and Properties of Matter.” 

Analysis of Seventh Grade LDE Questions for 
Two Units 

In this section, alignment studies that investigate the relationship 
among LDE questions and objectives under the standards of the 
“Structure and the Properties of Matter” and “Systems in Our 
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TABLE 3 — The dispersion of 7th grade LDE questions. 


Learning areas 

Chapters 

N 

% 

Physical Events 

Force and Motion 

2 

11.1 


Electricity in Our Life 

2 

11.1 


Light 

2 

11.1 

Earth and Universe 

Solar System and 
More: Space Puzzle 

1 

9.7 

Matter and Change 

The Structure and 
Properties of Matter 

5 

25 

Living Things 

Systems in Our Body 

4 

20.8 

and Life 

Human and Environment 

2 

11.1 


Total 

18 

100 


Body” units, were carried out by the ten reviewers according to 
the four alignment criteria. 

Criterion 1: Analysis of the LDE Questions for 
Categorical Concurrence — As seen in Table 4, five ques- 
tions were asked about the Structure and Properties of Matter. 
This unit has five standards and 55 objectives. In this section, the 
reviewers first examined the questions and then specifically 
focused on the objectives under standards which were directly 
related to the LDE questions. 

From Table 4, it can be seen that all reviewers gave hits for all 
the questions and only standard 5 did not take any hits. There was 
not any acceptable level of categorical concurrence between 
standard 2 and question 9. No question was asked in relation to 
standard 5. The science teachers slightly related questions 13 and 
14 with the objectives under standards 4 and 5. For example, 
question 14 is related to the objective 3.5 under standard 3 
and requires students to be able to explain the functions of the 


endocrine system using a model, board, or charts (STSE-4). Ques- 
tion 14 actually investigates whether students are able to show the 
endocrine system on the model, but students are not required to 
know the duties of the endocrine system. From the results, we can 
conclude that the content of the objectives under standards and the 
LDE questions examined were mostly met with a high agreement 
coefficient for the two units. 

Criterion 2: Analysis of the Questions for DOK 
Consistency — Under this criterion, the objectives under the 
standards for “Structure and the Properties of Matter” and 
“Systems in Our Body” units were compared with the nine LDE 
questions based on the complexity of knowledge required by each 
part. At the first stage, the distribution of the seventh grade level 
science and technology program, according to the objectives 
under the standards, as SPS, AV, STSE, and cognitive, were done 
according to their compatibility with the LDE questions. The 
DOK consistency of the seventh grade curriculum and the LDE 
questions for the curriculum objectives in terms of cognitive 
dimension were investigated in the second stage. 

From Table 5, the depth of knowledge for almost all of the 
questions is marked as suitable for the objectives under the stand- 
ards. From these findings, we could say that on the basis of the 
complexity of knowledge being demanded cognitively, the objec- 
tives under the standards and the LDE questions were consistent 
to each other in general. Therefore, the DOK consistency of the 
nine questions was marked as consistent. 

Criterion 3: Range of Knowledge Correspon- 
dence — In this process, each reviewer examined the LDE ques- 
tions and wrote down what types of knowledge or abilities 
students should have to be able to solve these questions and what 
each objective under the standard provides to students. 


TABLE 4 — Categorical concurrence of the 7"' grade LDE questions and the standards. 


Units 

Objectives 

Question 

Hits 

Concurrence 

Structure and the 

1 .2. They guess that different elements of atoms are different by using the models and figures 

8 

9/10 

Concurrent 

properties of matter 

(SPS-5, 6). a 

2.3. Shows nucleus, the core of the fundamental particles and electrons on the representative pictures. 

9 

5/10 

Non-concurren t 


3.6. To call the positively charged ions, “cation,” negatively charged ions, “anion” 

10 

8/10 

Concurrent 


4.2. Predict push/pull forces between ions, call the pull force “ionic bond” 

11 

7/10 

Concurrent 


6.5. Notice the increase in the speed of solubility with the increasing temperature. 

6.6. Discover that the speed of solubility increases when solvent’s particle shape is becoming smaller. 

12 

8/10 

Concurrent 

Systems in our body 

4.3. Explain which type of warnings does the sense organs take and how the answer is given 
(STSE-31, 32). b 

13 

7/10 

Concurrent 


3.5. Shows endocrine gland; on model, board and/or diagram and explain its function (STSE-4). c 

14 

7/10 

Concurrent 


1 .2. Students guess that the nutrients are required to change to become beneficial to the body. 

15 

4/10 

Non-concurren t 


2.1. Demonstrating structures and organs composing excretion system on model, board and/or 
schema (STSE-4). C 

2.2. Explain function and importance of excretion system. 

16 

9/10 

Concurrent 


a SPS 5 Determines the differences and similarities among subjects or events. SPS 6 Makes comparisons based on observations for one or more specifications. 
b STSE 3 1 Explains using the samples how developed technologies from the past until now change people’s individual and social life, work habits and interactions 
with the environment. STSE 32 Explains using samples that the specific scientific or technological improvement can have positive or negative, predictable or unpre- 
dictable effects on the individual, society, and environment. 

C STSE 4 Knows the importance of using models to develop and present scientific information. 
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TABLE 5 — Consistency of the 7 th grade LDE questions according to the curriculum outcomes. 


Unit 

Objectives 

Question 

Compatible 

Partially 

Weak 

Consistency 

Structure and the Properties of Matter 

Objective 1.2 

8 

8/10 

2/10 

- 

Consistent 


Objective 2.3 

9 

2/10 

2/10 

6/10 

Inconsistent 


Objective 3.6 

10 

8/10 

2/10 

- 

Consistent 


Objective 4.2 

11 

7/10 

2/10 

1/10 

Consistent 


Objective 6.5- 6.6 

12 

8/10 

1/10 

1/10 

Consistent 

Systems in Our Body 

Objective 4.3 

13 

7/10 

2/10 

1/10 

Consistent 


Objective 3.5 

14 

2/10 

2/10 

6/10 

Inconsistent 


Objective 1.2 

15 

1/10 

3/10 

6/10 

Inconsistent 


Objective 2. 1-2.2 

16 

7/10 

2/10 

1/10 

Consistent 


From Table 6, it can be seen that in nine questions belonging 
to two units, only four have acceptable knowledge correspon- 
dence with the objectives under the standards. Reviewers’ extra 
explanations about objectives and questions gave us valuable 
qualitative data about knowledge correspondence. The reviewers’ 
hits for each question support their explanations. Only four ques- 
tions’ knowledge correspondences were accepted as compatible. 
From these findings, we could say that a comparable span of 
knowledge expected of students by the objectives under the stand- 
ards for the two units is not the same as or does not correspond to 
the span of knowledge that students need to correctly answer the 
nine LDE questions. 


Criterion 4: Balance of Representation — in order to 
find how the questions asked were distributed among the objec- 
tives under the standards, a balance index calculation for each 
question was made. 

From Table 7, only two of the five questions were hit as accepta- 
ble for the balance of representation criteria, and only one question 
was hit as acceptable for the “Systems in Our Body” unit for the 
balance of representation criterion. This criterion shows that LDE 
questions were not distributed equally within objectives under the 
standards. This means that some objectives had more emphasis on 
the LDE questions than others. Therefore, many standards were 
ignored and acceptable balances were not established. 


TABLE 6 — Range of knowledge correspondence between questions and objectives. 


Objective 

Question 

Qualitative assessments of the results from the comparisons 

Hits 

Correspondence 

Objective 1.2 

8 

Recognition of elements in atoms through the given models. Subject content in terms of learning areas, 
objectives and standards are compatible. Students’ models must be compared to each other and similar 
ones must be categorized. In this context, question and objective appreciation is available. 

80% 

Full compliance 

Objective 2.3 

9 

Solution of the problem requires knowledge of the structure of the atom. Without knowing the concepts 
of protons, neutrons and electrons, it is not possible to solve the problem. Questions do not require any 
impression on the picture. 

40% 

Part compliance 

Objective 3.6 

10 

Knowing what is the concepts anion and cation is enough for finding the solution to the question. 

70% 

Full compliance 

Objective 4.2 

11 

What is the meaning of ionic bond for the solution to the question? Then, the atoms which are likely to 
take and give electrons should be detennined by counting the electrons at the outer layer. At the end, 
proper atom or atoms which can trade electron with the given atom should be determined. This last stage 
is in compliance with STS-9. 

80% 

Full compliance 

Objective 6.5 
Objective 6.6 

12 

The amount of sugar, liquid and type of liquid is held constant. Only the dimension of the particle and 
temperature of the liquid are changed. From this point of view, the question and the objective are 
coordinated. 

70% 

Full compliance 

Objective 4.3 

13 

For the solution to the question, just to have knowledge about two sense organs (skin and eye) is enough. 
Moreover, the reaction dimension given in the objective is also lacking. 

40% 

Part compliance 

Objective 3.5 

14 

Although the objective includes endocrine glands and hormones that they secrete, the question just 
mentioned concerns the hormone, not the endocrine gland responsible for the hormone. The question of 
subject content is partially coordinated with the learning area. There is not any relationship with scientific 
models. 

20% 

Weak 

Objective 1.2 

15 

Question only requires graph reading skills. 

20% 

Weak 

Objective 2.1, 
Objective 2.2 

16 

For the solution to the question, primarily excretion organs should really be determined on the given 
excretion system. This part of the question shows compliance with the objective; showing excretion 
organs on a visual material. In the solution to the question, the duties of the urinary bladder should also be 
known. But only the importance and functions of the excretion system are given in the objectives. Struc- 
ture and function of other organs are not mentioned with the statement, but only shown on the model. 

40% 

Part compliance 
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TABLE 7 — Balance index (BI) of selected LDE questions. 


Unit 

Question 

O a 

H b 

I c 

BI 

Representation 

Structure and the properties of matter 

8 

2 

3 

1 1,2 = 8,5 

0.67 

Weak 


9 

3 

5 

I 2 . 3.4 = 9,1,1 

0.4 

Unacceptable 


10 

2 

5 

1 1 .4 = 3,9 

0.3 

Unacceptable 


11 

4 

7 

1 1, 2,3,7 = 1,6, 1,2 

0.79 

Acceptable 


12 

3 

7 

1 1 , 2,3 = L3,7 

0.71 

Acceptable 

Systems in our body 

13 

5 

6 

12,3,5,6,8 = 5, 1,8, 4, 2 

0.2 

Unacceptable 


14 

2 

11 

I 6 , 7 = 8,10 

0.68 

Weak 


15 

3 

11 

12,3,4 = 10,4,6 

0.59 

Unacceptable 


16 

4 

11 

18 , 9 , 10,11 = 2,10,4,1 

0.73 

Acceptable 


a O = Total number of objectives hit for the standard 

b H = Total number of items hit for the standard 

C I (k) = Number of items hit corresponding to objective (k) 


Discussions and Implications 

When the science and technology curriculum was analyzed, it 
could be seen that in both levels the objectives under the standards 
are dominantly SPS and STSE. This finding does not support the 
idea that a positive attitude development towards science and 
technology courses should start from the very first stage of ele- 
mentary education. That is because developing scientific attitudes 
and values, as well as voluntary participation in science activities 
both at school and out of school, are the main targets for elemen- 
tary science and technology education [50], These weaknesses 
should be dealt with as soon as possible through necessary amend- 
ments to the existing science and technology program. 

SPS objectives are in the majority at the second stage of ele- 
mentary education, just as in the first stage of elementary educa- 
tion. Another interesting point is the SPS objectives increasingly 
become more weighted toward subject knowledge while the distri- 
bution of the objectives considered from sixth grade to eighth 
grade. Also, these objectives become more pronounced when 
compared with the STSE and AV objectives. At the end of ele- 
mentary education, the SPS objectives gain more importance and 
are taken into consideration more than the STSE and AV objec- 
tives. Another interesting point is that the AV objectives are the 
least seen type of objective at the second stage of elementary edu- 
cation. Some studies have shown that approximately 80 % of high 
school students in Turkey do not like science courses, and AV 
objectives are the least emphasized in the science and technology 
curriculum, which is one of the weakest points of the curriculum 
[9], AV objectives gain special importance because students are 
categorized at high school level and some will not encounter 
much science after elementary education. 

For a detailed analysis, we focused only on the seventh grade 
LDE questions because the sixth and eighth grade levels have 
similar attributes (the same commission prepares all LDE ques- 
tions in Turkey). At the seventh grade, the percentage dispersion 
of chapters and the number of questions asked on the LDE are as 
follows: Physical Events (33 %; 6), Earth and Universe (9.7%; 1), 
Matter and Change (25 %; 5), Living Things and Life (33 %; 6). 
From these findings, for the seventh grade, there is a balance 


among percentage dispersion of the learning areas and the 
proportion of the number of questions. However, this is a superfi- 
cial assessment. Examining by means of simply covering topics is 
never a good predictor of student achievement on standardized 
tests. If the evaluators use the coverage and cognitive emphasis to- 
gether, they should be in a better position to predict more mean- 
ingfully students’ performance on general tests [37,51]. 

When the balance between questions and the new science and 
technology program in the categorical concurrence criterion is an- 
alyzed, positive significant results are obtained. However, these 
results do not resemble the study of Saderholm and Tretter [21]. 
They analyzed similarities and differences between assessing 
structures of physics subjects such as Benchmarks for Science Lit- 
eracy (Benchmarks), National Science Education Standards 
(NSES), National Assessment Governing Board (NAEP), and 
Trends in International Mathematics and Science Study (TIMMS). 
These indicators are giving direction to science curricula all over 
the world. In their study, Saderholm and Tretter determined that 
physics topics had different rates for each of these four indicators, 
and some physics topics included in NAEP and TIMMS are not 
included in Benchmarks or NSES [21]. However, altering objec- 
tives for international comparative studies does not have any 
educational meaning unless the coverage of questions considered 
at national level. 

From the findings of the study, the vast majority of objectives 
and questions were coherent with the mental complexities of the 
LDE exam. Therefore, at the seventh grade level, LDE exam 
questions could meet the DOK criterion consistency in Webb’s 
alignment model [37]. Similarly, the Chinese curriculum also 
draws attention to subject knowledge and understanding, and 
gives importance to the skills stated previously, such as applica- 
tion and analysis, in examinations [5], Nationwide tests in Singa- 
pore require a higher level of thinking ability according to the 
standards, while there is a level of compliance between the exami- 
nation and standards cognitive levels in the United States [16]. In 
this context, Turkey shows similarity with the United States in 
terms of LDE examination and compatibility of programs accord- 
ing to the DOK consistency criteria. However, both objectives 
and questions, including low cognitive competencies, should not 
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be overlooked. The PISA evaluation requires different skills, 
which are induction, deduction, logical thinking, system-based 
thinking, and critical thinking. These are fundamental for the cog- 
nitive domain. In order to get a high score from the PISA evalua- 
tion, there ought to be a high level of thinking ability and all 
activities should be in line with this purpose. At this point, Turkey 
has made an improvement in that the LDE exam contains very 
little knowledge level questions. However, objectives should be 
reviewed again according to the PISA and TIMMS objectives if 
Turkey wants to improve its place in international rankings. 
Consequently, higher level questions such as analysis, synthesis, 
and evaluation should be given more space in the LDE examina- 
tions [52], 

Only three of nine questions were at the compatible level for 
being answered using knowledge described in the curriculum 
objectives when the questions were examined according to the 
range of knowledge correspondences criteria. In this respect, 
there is no compliance between the depth of knowledge in order 
to solve a question and the depth of knowledge that is envisaged 
in the standards. However, it can be claimed that examinations 
have to be adapted to schools rather than adapting schools 
according to the examinations [42], The entrance examinations, 
which are vital to the student’s future, are determined by the in- 
structive activities of the teacher. It might help teachers to 
enhance students’ ability to conceptualize rather than memorize 
by administering exams that are parallel to the outcomes and the 
topics of the curriculum. 

Although the questions in the LDE examination are distrib- 
uted according to each unit and there is a balance between the 
proportion of the standards and the number of questions, the bal- 
ance of representation criterion is mostly neglected. All objec- 
tives under the standards were not considered sufficiently in the 
question distribution of the LDE exam. The standards are seen as 
a general framework and objectives are regarded as more specific 
achievements that students are expected to attain. While the 
questions in the LDE examination were prepared so as to include 
standards, they did not fully cover the objectives. For example, if 
there are six objectives related to one standard, which is sup- 
posed to be addressed with one question, the LDE questions are 
mostly intended to assess one or two of these objectives. Only 
three of nine questions investigated in this study include a large 
number of objectives. In other words, a large number of objec- 
tives that students are expected to reach remained well beyond 
the scope of assessment in the examination. Thus, it should be 
kept in mind during question preparation that the LDE questions 
should have more than one objective. It is believed that the 
coherency between the curriculum and national tests shows the 
importance of the curriculum and also shows how successfully 
students acquire the objectives [14], Last but not least, should 
the LDE and the new curriculum have common characteristics; it 
would contribute to the assessment and improvement of the 
curriculum. 

There have been studies on the analysis of the LDE questions. 
These claimed that the objectives under the standards had great 
consistency with the LDE questions. On the basis of this claim, all 
policy makers argue that this current LDE is valid and reliable and 
at the same time it reflects the nature of the current science and 


technology curriculum [9], However, when we analyzed the LDE 
for this study by using the four criteria, we found that the ques- 
tions only fulfill the first two criteria and not the rest, which inves- 
tigate the close relationships between the curriculum and 
questions. In this case, we should re-assess all the exam questions 
for accurate conclusions. Assessment offices at the Ministry of 
National Education and some private schools are now looking for 
experts who are able to assess consistency among the curriculum 
guidelines and the questions asked in the exams. For now, there 
are only very few experts who are aware of the alignment issues 
and who are able to match the curriculum guidelines with the 
questions. 

Finally, it could be concluded from the findings and discussion 
that Turkey wants and plans to improve its ranking in international 
indicators such as PISA and TIMMS. Turkey’s willingness to 
improve in the educational area has already started to reflect in the 
PISA scores. According to the PISA results, the average perform- 
ance on the 2006 science literacy test was 10-points lower than 
the average score in 2003. This decrease should not be seen as a 
dichotomy because the decrease might come from difficulties with 
the adaptation process of the 2004 science and technology curricu- 
lum program. With the gradual adoption of the philosophies and 
application of the 2004 science and technology program, the aver- 
age science literacy score on the 2009 PISA for Turkey went up 
from 424 to 454 points. Although Turkey improved its PISA 
scores by 30 points, these scores still remain below the average 
science literacy point of OECD countries (compare 454 points 
with 501 points). At the same time, Turkey has continued to have 
a low place among the ranking of countries according to science 
literacy tests scores (36/41 in 2003, 44/57 in 2006, and 43/65 in 
2009) [53], In order to have a better ranking among participant 
countries in international examinations, Turkish policy makers 
should first analyze the contents and the nature of the questions 
asked in these international examinations. Second, all educators 
and researchers should be aware of the fact that the assessment 
framework of the international examination includes the concepts 
of scientific literacy; nature of science; basic science concepts; the 
relationships among science, technology, society and environ- 
ment; and the reflections of the fundamental objectives of SPS, 
STSE, and AV. If students grasp all these issues, they will be very 
successful at these international examinations in addition to using 
the knowledge gained in solving everyday life problems. Theoreti- 
cally, this might sound possible, but how exactly to put all these 
advanced ideas and goals into practice will remain as the key 
problem, which must be dealt with in the near future. We should 
keep in mind that raising achievement in terms of both internal 
and external indicators is a slow process that takes time. 
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Appendix 


Sample question from the level determination 
examination on science and technology. 

8. The teacher brought four cards to the classroom displaying the 
particle models of some objects and asked the students “Which 
objects can be made of the same matter according to particle 
models?”. 



Coffeepot 



Bracelet 


ooo 

ooo 

Q.QQ 



Pot 


• • • 
• • • 
• • • 



Vase 


Which of the following student responses is correct? 

A) Coffeepot-Vase 

B) Coffeepot-Pot 

C) Coffeepot-Pot- Vase 

D) Coffeepot-Bracelet-Vase 



Hi. my name is oxygen. I like to 
playgames. I want to play with 
the friends that can make ionic 
bond with me. Which of the 
friends helow can I play with? 




Argon 



Fluorine Magnesium 


A) Fluorine 

B) Magnesium 

C) Argon 

D) Argon and fluorine 
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